Content item allocation

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for content distribution. In one aspect, a method includes obtaining reservation data specifying a reservation period and a number of impression reservations for the reservation period; receiving, during the reservation period, content item request data specifying that a resource was requested; identifying candidate content item providers; determining an expected revenue that will be obtained by selecting one of the candidate content item providers to provide a content item; providing, to a content item exchange, auction data including a reserve price; receiving positive response data specifying that the content item exchange will provide an exchange content item for a price that meets the reserve price; and in response to receiving the positive response data, selecting the exchange content item as the content item to be provided in response to receiving the content item request data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.13/365,630, entitled “CONTENT ITEM ALLOCATION” filed Feb. 3, 2012, whichclaims the benefit under 35 U.S.C. §119(e) of U.S. Patent ApplicationNo. 61/439,469, entitled “YIELD OPTIMIZATION OF DISPLAY ADVERTISING WITHAD EXCHANGE,” filed Feb. 4, 2011. The entire contents of each of theforegoing applications are incorporated herein by reference.

BACKGROUND

This specification relates to content distribution.

The Internet has enabled access to a wide variety of resources, e.g.,video and/or audio files, web pages for particular subjects, and newsarticles. Such access to these resources has likewise enabledopportunities for providing relevant additional content. For example,resources of particular interest to a user can be identified by a searchengine in response to a user query. By comparing the user query to alist of keywords specified by an advertiser, it is possible to providerelevant advertisements to the user.

Another form of online advertising is advertisement syndication, whichallows advertisers to extend their marketing reach by distributingadvertisements to additional partners. For example, third partypublishers can place an advertiser's text or image advertisements on webpages that have content related to the advertisement. As the users arelikely interested in the particular content on the publisher webpage,they are also likely to be interested in the product or service featuredin the advertisement. Accordingly, such advertisement placement can helpdrive online customers to the advertiser's website.

In some situations, advertisements can be selected for placement on aweb page based on the outcome of an auction. For example, in ageneralized second price auction, the top n advertisers will be placedin the n available advertisement slots. In other situations, web pagepublishers enter into agreements with advertisers to allow displayadvertisements on their web pages for a certain period of time and/orfor a certain number of impressions, or views.

SUMMARY

This specification describes technologies relating to contentdistribution.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof obtaining reservation data for a resource that is provided by apublisher, the reservation data specifying a reservation period, and anumber of impression reservations for the reservation period, the numberof impression reservations being a number of resource requests for whichan ad slot of the resource has been reserved by one or more reservationcontent item providers; receiving, during the reservation period,content item request data specifying that the resource was requested;identifying one or more candidate content item providers from the one ormore reservation content item providers; determining an expected revenuethat will be obtained by selecting one of the candidate content itemproviders to provide a content item in response to receiving the contentitem request data; providing, to a content item exchange, auction dataincluding a reserve price that is based on the expected revenue;receiving positive response data from the content item exchange, thepositive response data specifying that the content item exchange willprovide an exchange content item for a price that meets the reserveprice; and in response to receiving the positive response data,selecting the exchange content item as the content item to be providedin response to receiving the content item request data. Otherembodiments of this aspect include corresponding systems, apparatus, andcomputer programs, configured to perform the actions of the methods,encoded on computer storage devices.

These and other embodiments can each optionally include one or more ofthe following features. The method may include the actions of receiving,during the reservation period, second content item request dataspecifying that the resource was requested; identifying one or moresecond candidate content item providers from the one or more reservationcontent item providers; determining a second expected revenue that willbe obtained by selecting one of the second candidate content itemproviders to provide a content item in response to receiving the secondcontent item request data; providing, to the content item exchange,second auction data including a second reserve price that is based onthe second expected revenue; receiving negative response data from thecontent item exchange, the negative response data specifying that thecontent item exchange will not provide an exchange content item for aprice that meets the second reserve price; and in response to receivingnegative response data, selecting a content item that is provided by oneof the second candidate content item providers as the content item to beprovided in response to receiving the second content item request data.

The method may also include the action of, in response to receivingnegative response data, selecting one of the second candidate contentitem providers as a winning content item provider, the winning contentitem provider being the second candidate content item providercorresponding to a highest impression score relative to all other secondcandidate content item providers; and selecting a content item that isprovided by one of the second candidate content item providers includesselecting a content item provided by the winning content item provider.

The method may also include the actions of, in response to receivingnegative response data: determining that two or more of the secondcandidate content item providers each correspond to a same impressionscore that is a highest impression score relative to all other secondcandidate content item providers, assigning each of the two or moresecond candidate content item providers an assignment probability basedon one or more characteristics associated with the resource request, andselecting one of the two or more second candidate content item providersbased on the assignment probabilities; and selecting a content item thatis provided by one of the second candidate content item providersincludes selecting a content item that is provided by the selectedsecond candidate content item provider.

The method may also include the actions of, in response to receivingnegative response data: determining that two or more of the secondcandidate content item providers each correspond to a same impressionscore that is a highest impression score relative to all other secondcandidate content item providers, assigning each of the two or moresecond candidate content item providers an assignment probability basedon a particular combination of the two or more second candidate contentitem providers that each correspond to the same impression score, andselecting one of the two or more second candidate content item providersbased on the assignment probabilities; and selecting a content item thatis provided by one of the second candidate content item providersincludes selecting a content item that is provided by the selectedsecond candidate content item provider.

The method may also include the action of determining that a number ofremaining resource requests for the reservation period exceeds a numberof unfulfilled impression reservations for the reservation period; andproviding auction data and receiving positive response data are onlyperformed in response to determining that the number of remainingresource requests exceeds the number of unfulfilled impressionreservations.

The reserve price may be determined based on an impression score for atleast one of the candidate content item providers, each impression scorespecifying, for a candidate content item provider, a measure of qualityof the impression to the candidate content item provider. Eachimpression score may be based on one or more characteristics associatedwith the content item request data, the one or more characteristicsincluding a predicted click-through rate. The reservation data mayfurther specify, for each of the one or more reservation content itemproviders, a portion of the impression reservations that have beenreserved for the reservation content item provider. Identifying one ormore candidate content item providers may include identifying areservation content item provider that has at least one unfulfilledimpression reservation. The reserve price may be generated based on afunction of the expected revenue and a probability of selection, and thecontent item to be provided in response to receiving the content itemrequest data may be an advertisement.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. Resource publishers are able to efficiently manageadvertising and/or other content item agreements while earning revenuethrough a content item auction while also providing relevantadvertisements or other third party content items to users accessingtheir resources.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example online environment in whichcontent is provided to a user.

FIG. 2 is a block diagram of an example data flow for selecting acontent item for presentation with a resource.

FIG. 3 is a flow diagram of an example process for selecting a contentitem to be provided to a user device.

FIG. 4 is a block diagram of an example data processing apparatus.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION Overview

A content item allocation system provides a user device with a contentitem in response to receiving a request for a resource (or content item)from the user device. The content item allocation system may provide acontent item from a content item provider who has reserved impressionsfrom a publisher that provides the resource. Alternatively, the contentitem allocation system may provide a content item from a content itemprovider who has bid a certain price for the impression through acontent item exchange. The methods and system described in thisspecification choose a content item to provide in a manner designed tofulfill content item reservations while potentially increasing ormaximizing revenue for the publisher providing the resource anddelivering more relevant content items to users.

In some implementations, the system receives reservation data for aresource. The reservation data specifies (i) a period of time, and (ii)a number of impression reservations reserved by reservation content itemproviders over the period of time. The system receives a request for theresource from a user device and identifies candidate content itemproviders from the reservation content item providers. If the estimatednumber of remaining impressions that will be received over the period oftime is greater than the number of unfulfilled impression reservationsfor the period of time, the system determines expected revenue that willbe obtained if one of the candidate content item providers is selectedto provide a content item, and then provides a content item exchangewith a reserve price that is based on the expected revenue.

The system then receives a response from the content item exchange, andthe response indicates whether the content item exchange received a bidthat meets or exceeds the reserve price. If the content item exchangereceives a bid greater than the reserve price, the system provides acontent item from the content item provider who submitted the winningbid (e.g., determined by a bid score based on quality and/or relevancesignals) to the exchange. If the content item exchange does not receivea bid greater than the reserve price, or if the number of remainingimpressions is not greater than the number of unfulfilled impressionreservations, the system provides a content item from one of thecandidate content item providers.

Example implementations are described in more detail in the sectionsthat follow.

Example Environment

FIG. 1 is an illustration of an example environment 100 in which contentis provided to a user. The example environment 100 includes a network102, such as a local area network (LAN), a wide area network (WAN), theInternet, or a combination thereof. The example environment 100 alsoincludes publishers 104 that host (directly or indirectly) resources106, user devices 108, content item providers 110, a content itemallocation system 112, a content item data store 114, and a content itemexchange 116. While the content item allocation system 112 and contentitem exchange 116 are depicted separately, they may be realized as partof a single system.

A resource 106 is data provided over the network 102. A resource 106 isidentified by a resource address that is associated with the resource106. Resources 106 include HTML pages, word processing documents,portable document format (PDF) documents, images, video, and feedsources, to name only a few. The resources 106 can include content, suchas words, phrases, images and sounds, that may include embeddedinformation (such as meta-information in hyperlinks) and/or embeddedinstructions (such as scripts). Resources 106 may also include one ormore ad slots 118, which are portions of a resource 106 in whichadvertisements or other content items are inserted.

Resources are generally hosted by publishers 104 either directly (e.g.,on a server computer owned by the publisher) or indirectly (e.g., on aserver computer owned by a third party but paid for by the publisher). Apublisher 104 is an entity that hosts and/or provides electronic accessto a resource (e.g., web page content) by use of the network 102.Publishers 104 may generate revenue by selling content item space (e.g.,ad slots 118) to content item providers 110.

Content item providers 110 are entities, such as advertisers (and mayalso be publishers), that provide content items, such as advertisements,for display with resources 106. The content items may be any type ofdata, and can include, for example, informational, education, orentertainment content, as well as advertisements in the form ofgraphical advertisements, such as banner advertisements, text onlyadvertisements, image advertisements, audio advertisements, videoadvertisements, advertisement gadgets with or without interactivefeatures, advertisements combining one of more of any of suchcomponents, etc., or any other type of electronic advertisement orcontent item document. The content items may also include embeddedinformation, such as a links, meta-information, and/or machineexecutable instructions, such as HTML or JavaScript™.

A user device 108 is an electronic device that is under control of auser and is capable of requesting and receiving resources 106 over thenetwork 102. Example user devices 108 include personal computers, mobilecommunication devices, and other devices that can send and receive dataover the network 102. A user device 108 typically includes a userapplication, such as a web browser, to facilitate the sending andreceiving of data over the network 102.

The content item allocation system 112 may be comprised of one or moredata processing apparatus, is capable of communicating across thenetwork 102, and may be coupled to a content item data store 114. Thesystem 112 provides content items for display with resources 106. Thesystem 112 may be in direct or indirect communication with publishers104 and content item providers 110 to manage publishers' content iteminventory and content item providers' demands.

The content item data store 114 may store information related to thepublishers 104 and content item providers 110. For example, the contentitem data store 114 may include, for a publisher, information about thecontent of resources 106 provided by the publisher, available ad slots118, targeting rules or restrictions of the ad slots 118, andreservation data (if applicable) for ad slots reserved by one or morecontent item providers. The content item data store 114 may alsoinclude, for a content item provider, the content item provider'scontent items (e.g., advertisements) and campaign data for a contentitem provider indicating budgets, selection criteria, and otherinformation relevant to a content item provider's campaign.

The content item allocation system 112 may also be coupled to a contentitem exchange 116 that conducts content item auctions for available adslots 118. Content item providers 110 may be permitted to select, orbid, an amount the content item providers 110 are willing to pay foreach impression of a content item, e.g., a cost-per-mille amount anadvertiser pays for presentation of an advertisement (e.g., acost-per-thousand impressions) or an amount that the content itemprovider pays when, for example, a user clicks on an advertisement(e.g., a cost-per-click). The cost-per-click can include a maximumcost-per-click, e.g., the maximum amount an advertiser is willing to payfor each click of an advertisement based on a keyword that is used toselect the advertisement. For example, advertisers A, B, and C allselect, or bid, a maximum cost-per-click of $0.50, $0.75, and $1.00,respectively. The content item exchange 116 can select the advertisementprovided by the highest bidding advertiser and provide thatadvertisement for display in the auctioned ad slot 118. In someimplementations, quality and/or relevance signals are used to determinea bid score, and the advertisement with the highest bid score can beselected for display.

When a user device 108 submits a resource request 120 to a publisher104, the publisher 104 causes the requested resource 122 to be providedto the user device 108 in response to the request 120. The requestedresource 122 content can include executable instructions that can beexecuted at the user device 108 to request content items from thecontent item allocation system 112. When the content item allocationsystem 112 receives a request for a content item, it may (i) provide acontent item from a content item provider 110 who has an agreement withthe publisher 104 to provide a content item for the requested resource122, (ii) provide a content item obtained from the content item exchange116, or (iii) provide no content item. In some implementations, thecontent item allocation system 112 chooses the option that is designedto provide the most revenue for the publisher, while also fulfilling allcontent item reservations. Further detail regarding the operation of thecontent item allocation system 112 is provided in the paragraphs thatfollow.

FIG. 2 is a block diagram of an example data flow 200 for selecting acontent item for presentation with a resource. The content itemallocation system 112 obtains reservation data 204 for a resourceprovided by a publisher 202. In some implementations, the reservationdata 204 is provided to the content item allocation system 112 by thepublisher 202 and/or content item providers who reserve impressions. Insome implementations, the reservation data 204 specifies a reservationperiod—e.g., a day, week, or month—and a total number of impressionreservations for the reservation period. The total number of impressionreservations is a number of resource requests for which an ad slot ofthe resource has been reserved by one or more reservation content itemproviders. Reservation data may be provided for any number of resourcesprovided by the publisher 202.

For example, the publisher of a sports news web site may have a web pagededicated to baseball news, and that web page may include an ad slot.The publisher may have entered into an agreement with an advertiser whoruns an online baseball equipment store, and the agreement might specifythat the publisher has agreed to serve the advertiser's advertisement inthe ad slot 1,000 times per week for 6 weeks. The agreement may specifyother information, such as selection criteria (e.g., restrictions onwhich ads can be served to what type of web site visitor at what time ofday, etc.), particular advertisements to be served (if more than one),and/or price(s) to be paid. The price(s) may be a lump sum, a price perthousand impressions, a price per click, a variable price (e.g., onethat depends on satisfaction of one or more selection criteria), or anyother appropriate price or combination of prices.

The reservation data for the baseball news web page may include any ofthe foregoing information, but it at least includes a reservation period(e.g., a week), and a number of impression reservations for that week(e.g., 1,000 in the foregoing example). The reservation period that isreferred to throughout this document may or may not match time periodsset forth in an agreement between the publisher and content itemprovider. For example, an advertiser might enter into a month-longagreement to reserve 1 million impressions, while the reservation periodmay be a proper subset of that period (e.g., 1 day). The reservationdata for the baseball news web page would also include impressionreservation data for any other advertisers who have agreed to provideadvertisements in the example ad slot—or any other ad slots on the webpage.

In some implementations, a user device 206 sends a resource request 208to the publisher 202 and receives the requested resource 210 in return.The requested resource 210 may cause the user device 206 to transmit acontent item request 212 to the content item allocation system 112. Forexample, the requested resource 210 may include code (e.g., one or morescripts) that, when executed by the user device 206, causes the userdevice 206 to issue a content item request 212 to the content itemallocation system. The content item request 212 may include anyinformation relevant to the provision of a content item—e.g., therequested resource and user data that can be used to determine ifapplicable selection criteria are met. In some implementations, thepublisher 202, after receiving the resource request 208, may send acontent item request to the content item allocation system 112, receivea content item in return, and provide both the requested resource andthe content item to the user device 206. Such an implementation wouldnot require the user device 206 to initiate the content item request.

The content item allocation system 112 identifies candidate content itemproviders 214 from the reservation content item providers 216 that werespecified by the reservation data 204. In some implementations, thereservation data 204 specifies the portion of the total impressionreservations that have been reserved by each reservation content itemprovider 216 and the number of unfulfilled impression reservations(i.e., the number of impression reservations that still need to befulfilled during the reservation period) for each reservation contentitem provider 216. In some implementations, a reservation content itemprovider 216 is only identified as a candidate content item provider ifit has at least one unfulfilled impression reservation.

For example, the reservation data for a baseball news web page mayindicate that there are three reservation advertisers for thatpage—advertiser1, advertiser2, and advertiser3. The reservation data mayfurther specify the number of unfulfilled impression reservations foreach advertiser for the reservation period. If the reservation period isone day and advertised has reserved 100 advertisement impressions forthat day, 90 of them may have already been fulfilled, leaving 10reserved impressions unfulfilled for the day. Similarly, advertiser2 mayhave 5 unfulfilled impressions, and advertiser3 may have 0 unfulfilledimpressions (indicating that advertiser3's reserved impressions havealready been fulfilled). In some implementations, advertiser1 andadvertiser2 will be identified as candidate advertisers based on each ofthem having at least one unfulfilled impression. Because advertiser3 hasno unfulfilled impressions (e.g., its reserved impressions have alreadybeen fulfilled or are 0 for the reservation period), advtertiser3 willnot be identified as a candidate advertiser.

In some implementations, the content item allocation system 112determines whether an estimated number of remaining resource requestsfor the reservation period exceeds a number of unfulfilled impressionreservations for that period. For example, the reservation data for aresource may specify the expected number of resource requests for thereservation period—e.g., the publisher of the baseball news web page mayprovide an estimated number of visitors to the web page per day based onhistorical data. The estimated number of remaining resource requests maybe based on, for example, historical resource requests logged by thepublisher, the content item allocation system, or a third party. Theexpected number of resource requests may be compared to the total numberof unfulfilled impressions reservations for the reservation period.

In response to determining that the number of remaining resourcerequests does not exceed the number of unfulfilled impressionreservations, the content item allocation system 112 may provide acontent item from one of the candidate content item providers withoutrequesting a content item from the content item exchange 116. Forexample, submitting auction data 218 (discussed in further detail below)to the content item exchange 116 may be conditioned on there being moreremaining resource requests than unfulfilled impression reservations.This condition is designed to prevent a publisher from failing tofulfill the reservation content item providers' impression reservations.In response to determining that the number of remaining resourcerequests exceeds the number of unfulfilled impression reservations, thecontent item allocation system 112 continues the process of selecting acontent item to be displayed with the requested resource.

The content item allocation system 112 determines an expected revenuethat will be earned by selecting one of the candidate content itemproviders 214 to provide a content item to the user device 206 inresponse to the content item request 212. In some implementations, theexpected revenue is simply the amount the candidate content itemprovider 214 has agreed to pay for a reserved impression. For example,advertiser) may have agreed to pay $500 for a thousand impressions,which results in $0.50 per impression, and advertiser2 may have agreedto pay $400 for a thousand impressions, which results in $0.40 perimpression. The expected revenue may be $0.50 (based on the highestamount offered by a candidate advertiser), or some other amount, such as$0.45 (an average of the amounts offered by the candidate advertisers)or $0.40 (the lowest amount offered by a candidate advertiser).Generally, in order to achieve a threshold revenue (e.g., a highestrevenue) for the publisher, the expected revenue will be based on thehighest amount offered by a candidate advertiser ($0.50 in thisexample).

In some implementations, the expected revenue is determined based onother factors, such as an impression score for one or more of thecandidate content item providers. An impression score specifies, for acandidate content item provider, a measure of quality of the impressionreservation. An impression score may be candidate content item providerspecific, and increase or decrease based on characteristics associatedwith the content item request. One such characteristic is a predictedclick-through rate (pCTR) of an impression. The pCTR of an impression isan estimated likelihood that a user will click on a content item. pCTRsmay be based on many factors, such as a quality associated with theparticular resource being presented (e.g., a historic click-through ratefor all content items presented with the resource), the content item,the content item provider, the content item campaign, and the user (orusers). The quality of the foregoing factors may be based on pastperformance of one or more of the factors. As described in more detailbelow, the impression score of a particular impression for a particularreservation content item provider can also be based on a capacityconstraint indicative of a portion of all available impressions thathave been reserved by the particular reservation content item provider.

The impression score may affect the expected revenue if a candidatecontent item provider's offer for an impression depends on the score.For example, candidate advertiser1 may specify that the amount it iswilling to pay for a reserved impression depends on the pCTR of theimpression, e.g., by offering a product of the offer ($0.50) and thepCTR of the impression. Thus, if the pCTR for a given impression is 60%(e.g., there is a 60% chance that this impression will be clicked on bythis user), advertiser1's offer of $0.50 may be decreased to $0.30($0.50*0.6=$0.30). Many suitable factors may be considered incalculating an impression score or pCTR, and many suitable formulaeexist for creating a candidate content item provider's dependency onimpression score or pCTR.

The impression score for a particular impression may vary on aper-content item provider basis and be based on selection criteriaspecified by the content item provider. In some implementations, theimpression score for a particular impression may be directlyproportional to a level of match between content item provider selectioncriteria and characteristics of the impression (e.g., demographic andgeographic information for the user to which an advertisement will bepresented). Thus, as the level of match between a content itemprovider's selection criteria and characteristics of the impressionincreases, so will the impression score for that impression with respectto that content item provider.

The impression score can be considered an indication of the value of aparticular impression, such that a content item provider may be willingto pay more for impressions having higher impression scores, which mayincrease the expected revenue for that impression. For example, thereservation data 204 may indicate that advertiser2 offers a variableprice for an impression based on selection criteria that are met by dataincluded in the content item request 212, such as user profileinformation. Accordingly, advertiser2 may be willing to pay more for animpression that satisfies its selection criteria, such as a user'sgeographic location. If the user profile associated with the contentitem request 212 indicates that a user is in the United States,advertiser2 may increase its offer for the particular impression by$0.20, and if the user is in the Eastern time zone, advertiser2 mayincrease its offer for the particular impression by $0.15. Thus, if theuser submitting the content item request 212 is in the United States andin the Eastern time zone, the expected revenue earned by selectingadvertiser2 may be $0.75 ($0.40+$0.20+$0.15=$0.75). Many other suitablefactors and selection criteria exist and may be considered indetermining an expected revenue.

In some implementations, a user may choose to opt-in to and/or opt-outof the inclusion of user data in the content item request. For example,if a user opts into the inclusion of user data with the content itemrequest, the content item allocation system can use the user data toprovide content items that may be relevant to that user. If a userchooses to opt-out of the inclusion of user data with the content itemrequest, the content item allocation system does not use user data whenselecting a content item to be provided. In some implementations, userdata is anonymized to protect privacy.

The content item allocation system 112 provides auction data 218 to thecontent item exchange 116, and the auction data 218 includes a reserveprice that can be based on the expected revenue for a particularimpression. In some implementations, the reserve price included in theauction data is the expected revenue or the expected revenue plus $0.01.For example, if the expected revenue is $0.50 (e.g., the expectedrevenue earned by selecting candidate advertiser, advertised), thereserve price may be $0.51. This reserve price represents the minimumprice that will be accepted from a content item provider bidding for theimpression through the content item exchange 116. The auction data 306may also include other data of interest to potential content itemproviders bidding for the impression, such as user data that wasincluded in the content item request 212 or data specifyingcharacteristics of the requested resource 210. Other methods ofdetermining that reserve price can also be used, as described in moredetail below.

In some implementations, the content item allocation system 112 obtainsa reserve price that is generated based on the expected revenue and aprobability of selection. The probability of selection is theprobability that an available impression will receive a winning bid at agiven price. For example, if an available impression is auctioned for$1.00, the probability that a content item provider will bid that amountmay be 80%. The probability of selection may be based on many factors,such as the price, and/or a quality of the resource, publisher, or userdata associated with the available impression. The probability ofselection may be based on historical data, and may be calculated by thecontent item allocation system 112, the content item exchange 116, orany other suitable third party.

Relationship (1) provided below can be used to generate an auctionreserve price based on an opportunity cost and the probability ofallocating the impression through a content item exchange. Theopportunity cost represents what a publisher 202 stands to lose if theavailable impression is not awarded to a reservation content itemprovider 214. For example, the opportunity cost may be equal to theexpected revenue that the publisher 202 will receive by awarding theimpression to a reservation content item provider. The auction reserveprice may be based on the solution to the following example problem:

$\begin{matrix}{{{R(c)} = {{\max\limits_{p \geq 0}{{F(p)}p}} + {\left( {1 - {F(p)}} \right)c}}};} & (1)\end{matrix}$

where,

c is the opportunity cost of not allocating the impression to areservation content item provider;

R(c) is the estimated revenue given opportunity cost c;

F(p) is the probability of allocating the impression through the contentitem exchange at a price of p.

Relationship (1) is solved to identify the price p at whichF(p)p+(1−F(p))c is maximized. If c=expected revenue=$0.50, solving theabove problem will result in the maximum estimated revenue R(c), and theprice p that solves the problem becomes the auction reserve price. Forexample, if F($1.00) is 0.5 (e.g., the probability that a winning bidwill be received if the impression is auctioned with a price of $1.00 is50%), relationship (1) results in an estimated revenue value of $0.75(0.5*$1.00+(1-0.5)*$0.50=$0.50+$0.25=$0.75). By way of further example,if F($0.80) is 0.9, the function results in an estimated revenue valueof $0.77 (0.9*$0.80+(1-0.9)*$0.50=$0.72+$0.05=$0.77). If the estimatedrevenue value of $0.77 is the maximum estimated revenue value, then theauction reserve price is the price p ($0.80) that maximizes the function(e.g., the value that solves the above problem). The numbers provided inthe preceding example are for purposes of illustration only, andaccording to relationship (1) the price p that would be the value of pthat maximizes R(c).

Relationship (1) and the discussion above provide one example of amethod for generating an auction reserve price based on the expectedrevenue and probability of selection. In some implementations, otherfactors are considered in generating the auction reserve price. Forexample, user profile information or pCTR associated with the availableimpression can be used to adjust the auction reserve price for aparticular impression. Any other suitable factors relevant toprobability of selection may be used to generate the auction reserveprice.

Additionally, the impression scores for a particular impression may beused to adjust the reserve price for the particular impression. In someimplementations, the price p computed above can be scaled using theimpression score, as illustrated by relationship 2.

reserve price=p(max_(aεAn){0,Q _(n,a) −v _(a)})  (2)

where,

reserve price is the minimum price that an content item provider mustpay to obtain impression n through the content item exchange;

p is the price p that maximizes relationship 1 for opportunity cost c,e.g., p(c)=max_(aεAn){0, Q_(n,a)−v_(a)};

A_(n) denotes the set of content item providers that have unfulfilledimpressions remaining (e.g., the set of candidate content itemproviders)

n denotes a particular impression;

{Q_(n,a)−v_(a)} is the impression score of impression n for content itemprovider a;

Q_(n,a) is a measure of impression quality for impression n to contentitem provider a; and

v_(a) is a value indicative of a capacity constraint for content itemprovider a.

In some implementations, the value v_(a) can be obtained for eachreservation content item provider using relationship (3).

Min_(va) {E[R(max{Q _(a) −v _(a)})+Σ_(aεA) v _(a)ρ_(a)]}  (3)

where,

the expectation E is taken of random impressions according to somedistribution. In some implementations, the estimate of this distributionmay be obtained using historical data;

R is the expected revenue for a set of impressions to be received over atime period; and

ρ_(a) is the number of reservation impressions for content item providera relative to a total number of impressions in the set of impressions.

Once the reserve price has been determined, this reserve price isprovided to the content item exchange 116 with the auction data 218, andthe content item exchange 116 can attempt to allocate the impression ata price that is higher than the reserve price. For example, the contentitem exchange 116 may receive one or more bids 222 from one or moreauction content item providers 224 who wish to provide an exchangecontent item (i.e., a content item that is distributed by the contentitem exchange 116) for the available impression. The content itemexchange 116 may select a winning content item provider based on thebids 222 (e.g., the highest bidding auction content item provider may beselected as the winning content item provider).

If the bid submitted by the winning content item provider meets thereserve price, the content item exchange 116 provides positive responsedata to the content item allocation system 112. For example, if theauction data 218 specifies a reserve price of $0.80, the content itemexchange 116 conducts an auction with a minimum bid of $0.80. If a bidis received that is greater than $0.80, the content item exchange 116provides positive response data to the content item allocation system112. In some implementations, the positive response data specifies theexchange content item provided by the winning content item providerand/or the winning bid.

If the content item exchange 116 is unable to allocate the impression ata price that exceeds the auction reserve price, the response data 220provided by the content item exchange 116 is negative response data thatspecifies that the content item exchange 116 will not provide anexchange content item for a price that meets the reserve price. Forexample, if the content item exchange 116 conducts an auction, but failsto receive a bid greater than the reserve price, the content itemexchange 116 provides negative response data to the content itemallocation system 112.

In response to receiving positive response data, the content itemallocation system 112 selects the exchange content item as the contentitem 226 to be provided in response to the content item request 212. Insome implementations, the content item allocation system 112 providesthe user device 206 with the content item 226. In some implementations,the content item allocation system 112 provides the user device 206 withdata that causes the user device 206 to obtain and display the contentitem 226. For example, the content item allocation system 112 mayprovide the user device 206 with code that, when executed by the userdevice 206, obtains the content item from a content item provider (e.g.,the winning content item provider).

In response to receiving negative response data, the content itemallocation system 112 selects a content item that is provided by one ofthe candidate content item providers 214 as the content item 226 to beprovided in response to the content item request 212. In someimplementations, the content item allocation system 112 selects thecandidate content item provider with the highest impression scorerelative to all other candidate content item providers as a winningcontent item provider, and selects a content item that is provided bythe winning content item provider as the content item 226 to be providedin response to the content item request 212.

In some implementations, in response to receiving negative responsedata, the content item allocation system 112 may determine that two ormore candidate content item providers 214 both have the highestimpression score for the impression being allocated. In this situation,the two or more candidate content item providers are considered to havetied in the allocation process. Many suitable methods for breaking a tieexist, and some example methods for breaking ties are described in theparagraphs that follow.

In some implementations, each of the candidate content item providershaving the highest impression score is assigned an assignmentprobability based on one or more characteristics associated with thecontent item request, such as a pCTR. For example, if candidateadvertised offers an advertisement with a pCTR of 0.6 (e.g., a 60%chance that its advertisement will be clicked on by the user associatedwith the user device 206), and candidate advertiser2 offers anadvertisement with a pCTR of 0.5 (e.g., a 50% chance that itsadvertisement will be clicked on by the user associated with the userdevice 206), the assignment probability for candidate advertised may be0.545 (0.6/(0.6+0.5)=0.6/1.1=0.545, or 54.5%), and the assignmentprobability for candidate advertiser2 may be 0.455(0.5/(0.6+0.5)=0.5/1.1=0.455, or 45.5%). In turn, the impression can bepseudo-randomly assigned to one of advertiser1 and advertiser2, wherethe assignment is constrained by the assignment probabilities. Anysuitable content item request characteristics and/or formula may be usedto calculate the assignment probability for each candidate content itemprovider.

In some implementations, each of the candidate content item providersthat had the same highest impression score is assigned an assignmentprobability based on the number of unfulfilled impressions for each ofthe tied candidate content item providers. For example, if candidateadvertiser1 has 10 unfulfilled impression reservations to fulfill, andcandidate advertiser2 has 5 unfulfilled impression reservations tofulfill, the assignment probability may be based on the ratio of theirunfulfilled impressions. The assignment probability for candidateadvertised may be 0.66 (10/(10+5)=10/15=0.66, or 66%), and theassignment probability for candidate advertiser2 may be 0.33(5/(10+5)=5/15=0.33, or 33%). Any suitable data associated with thecandidate content item providers and/or formula may be used to calculatethe assignment probability for each candidate content item provider.

In some implementations, a candidate content item provider is selectedto receive the impression based on the assignment probabilities. In thefirst example above, the assignment probabilities indicate thatcandidate advertiser1 has a 54.5% chance of being selected, andcandidate advertiser2 has a 45.5% chance of being selected. In thesecond example above, the assignment probabilities indicate thatcandidate advertiser1 has a 66% chance of being selected, and candidateadvertiser2 has a 33% chance of being selected. Any suitable method ofselection based on the probabilities may be used. For example, theimpression can be allocated pseudo-randomly as constrained by theprobabilities, such that a particular allocation is pseudo-random, whilethe allocation of multiple impressions is constrained by theprobabilities. In some implementations, the assignment probabilities ofeach candidate content item provider are chosen such that the totalexpected impressions that a reservation content item provider receivesin this way is equal to the number of impressions reserved by thatcontent item provider.

FIG. 3 is a flow diagram of an example process 300 for selecting acontent item to be provided to a user device. The process 300 may beperformed by one or more data processing apparatus of the content itemallocation system 112.

Reservation data is obtained for a resource (302). In someimplementations, the resource is provided by a publisher, and thereservation data specifies (i) a reservation period; and (ii) a numberof impression reservations being a number of resource requests for whichan ad slot of the resource has been reserved by one or more reservationcontent item providers. In some implementations, the reservation datafurther specifies, for each reservation content item provider, a portionof the impression reservations that have been reserved for thatreservation content item provider.

Content item request data indicating that the resource was requested isreceived (304). In some implementations, the content item request datais received from a user device. In some implementations, the contentitem request data is received from a publisher.

One or more candidate content item providers are identified (306). Insome implementations, the one or more candidate content item providersare identified from the reservation content item providers. For example,a reservation content item provider may only be identified as acandidate content item provider if it has at least one unfulfilledimpression reservation.

An expected revenue that will be obtained by selecting one of thecandidate content item providers is determined (308). In someimplementations, the expected revenue is the amount that will beobtained by providing a content item from one of the candidate contentitem providers in response to the content item request. In someimplementations, the expected revenue may be determined based on animpression score for one of the candidate content item providers, theimpression score specifying a measure of quality of the impressionreservation with respect to the candidate content item provider. Theimpression score may be based on one or more characteristics associatedwith the content item request (and/or the resource request). Forexample, one of the characteristics may be a predicted click-throughrate.

Auction data is provided to the content item exchange (310). In someimplementations, the auction data includes a reserve price that is basedon the expected revenue. In some implementations, the reserve price isgenerated based on a function of the expected revenue and a probabilityof selection, the probability of selection indicating the likelihoodthat an available impression would receive a bid on a content itemauction. In some implementations, the reserve price is based on theimpression score associated with one of the candidate content itemproviders. In another implementation, the auction data is only providedin response to determining that a number of remaining resource requestsfor the reservation period exceeds a number of unfulfilled impressionsreservations for the reservation period.

Response data is received from the content item exchange (312). In someimplementations, the response data indicates whether or not the contentitem exchange will provide an exchange content item for a price thatmeets the reserve price. In some implementations, response data is notreceived. For example, if the number of remaining resource requests forthe reservation period does not exceed the number of unfulfilledimpression reservations, the response data may not be received.

Whether the response data is positive response data or negative responsedata is determined (314). In response to receiving positive responsedata, the exchange content item is selected as the content item to beprovided in response to the content item request (316). In response toreceiving negative response data, the content item that is provided byone of the candidate content item providers is selected as the contentitem to be provided in response to the content item request (318).

In some implementations, the process 300 may loop back to step 304 aftercompletion of step 316 or 318. In this manner, multiple content itemrequests can be received, and multiple content items may be selected foreach of a plurality of resource requests.

FIG. 4 is a block diagram of an example data processing apparatus 400that can be used to perform operations described above. The apparatus400 includes a processor 410, a memory 420, a storage device 430, and aninput/output device 440. Each of the components 410, 420, 430, and 440can be interconnected, for example, using a system bus 450. Theprocessor 410 is capable of processing instructions for execution withinthe system 400. In one implementation, the processor 410 is asingle-threaded processor. In another implementation, the processor 410is a multi-threaded processor. The processor 410 is capable ofprocessing instructions stored in the memory 420 or on the storagedevice 430.

The memory 420 stores information within the apparatus 400. In oneimplementation, the memory 420 is a computer-readable medium. In oneimplementation, the memory 420 is a volatile memory unit. In anotherimplementation, the memory 420 is a non-volatile memory unit.

The storage device 430 is capable of providing mass storage for theapparatus 400. In one implementation, the storage device 430 is acomputer-readable medium. In various different implementations, thestorage device 430 can include, for example, a hard disk device, anoptical disk device, a storage device that is shared over a network bymultiple computing devices (e.g., a cloud storage device), or some otherlarge capacity storage device.

The input/output device 440 provides input/output operations for theapparatus 400. In one implementation, the input/output device 440 caninclude one or more of a network interface devices, e.g., an Ethernetcard, a serial communication device, e.g., and RS-232 port, and/or awireless interface device, e.g., and 802.11 card. In anotherimplementation, the input/output device can include driver devicesconfigured to receive input data and send output data to otherinput/output devices, e.g., keyboard, printer and display devices 460.Other implementations, however, can also be used, such as mobilecomputing devices, mobile communication devices, set-top box televisionclient devices, etc.

Although an example data processing apparatus has been described in FIG.4, implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in other types ofdigital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media (e.g., multiple CDs, disks, orother storage devices).

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular embodiments of particular inventions.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is: 1-36. (canceled)
 37. A method performed by one ormore data processing apparatus, the method comprising: receiving, by adata processing apparatus and during a reservation period, a request fora resource that includes a content item slot that has been reserved byone or more reservation content item providers; determining, by the dataprocessing apparatus, an expected revenue that will be obtained byselecting one of the reservation content item providers to provide acontent item in response to receiving the content item request data; andproviding, by the data processing apparatus and to a content itemexchange, a reserve price that is based on the expected revenue.
 38. Themethod of claim 37, further comprising: receiving positive response datafrom the content item exchange, the positive response data specifyingthat the content item exchange will provide an exchange content item fora price that meets the reserve price; and in response to receiving thepositive response data, selecting the exchange content item as thecontent item to be provided in response to receiving the request. 39.The method of claim 37, further comprising: receiving negative responsedata from the content item exchange, the negative response dataspecifying that the content item exchange will not provide an exchangecontent item for a price that meets the reserve price; and in responseto receiving negative response data, selecting a content item that isprovided by one of the reservation content item providers as the contentitem to be provided in response to receiving request.
 40. The method ofclaim 39, further comprising: in response to receiving negative responsedata, selecting one of the reservation content item providers as awinning content item provider, the winning content item provider beingthe reservation content item provider corresponding to a highestimpression score relative to all other reservation content itemproviders; and wherein selecting a content item that is provided by oneof the reservation content item providers comprises selecting a contentitem provided by the winning content item provider.
 41. The method ofclaim 39, further comprising: in response to receiving negative responsedata: determining that two or more of the reservation content itemproviders each correspond to a same impression score that is a highestimpression score relative to all other reservation content itemproviders; assigning each of the two or more reservation content itemproviders an assignment probability based on one or more characteristicsassociated with the request; and selecting one of the two or morereservation content item providers based on the assignmentprobabilities; and wherein selecting a content item that is provided byone of the reservation content item providers comprises selecting acontent item that is provided by the selected reservation content itemprovider.
 42. The method of claim 39, further comprising: in response toreceiving negative response data: determining that two or more of thereservation content item providers each correspond to a same impressionscore that is a highest impression score relative to all otherreservation content item providers; assigning each of the two or morereservation content item providers an assignment probability based on aparticular combination of the two or more reservation content itemproviders that each correspond to the same impression score; andselecting one of the two or more reservation content item providersbased on the assignment probabilities; and wherein selecting a contentitem that is provided by one of the reservation content item providerscomprises selecting a content item that is provided by the selectedreservation content item provider.
 43. The method of claim 37, furthercomprising: determining that a number of remaining resource requests forthe reservation period exceeds a number of unfulfilled impressionreservations for the reservation period; and wherein the providing areserve price is only performed in response to determining that thenumber of remaining resource requests exceeds the number of unfulfilledimpression reservations.
 44. The method of claim 37, wherein the reserveprice is determined based on an impression score for at least one of thereservation content item providers, each impression score specifying,for a reservation content item provider, a measure of quality of theimpression to the reservation content item provider.
 45. The method ofclaim 44, wherein each impression score is based on one or morecharacteristics associated with the request, the one or morecharacteristics including a predicted click-through rate.
 46. The methodof claim 37, wherein each reservation content item provider for which anexpected revenue is obtained has at least one unfulfilled impressionreservation.
 47. The method of claim 37, wherein the reserve price isgenerated based on a function of the expected revenue and a probabilityof selection.
 48. A system comprising: a data processing apparatus; anda data store storing instructions that, upon execution, cause the dataprocessing apparatus to perform operations comprising: receiving, duringa reservation period, a request for a resource that includes a contentitem slot that has been reserved by one or more reservation content itemproviders; determining an expected revenue that will be obtained byselecting one of the reservation content item providers to provide acontent item in response to receiving the content item request data; andproviding to a content item exchange, a reserve price that is based onthe expected revenue.
 49. The system of claim 48, wherein the operationsfurther comprise: receiving positive response data from the content itemexchange, the positive response data specifying that the content itemexchange will provide an exchange content item for a price that meetsthe reserve price; and in response to receiving the positive responsedata, selecting the exchange content item as the content item to beprovided in response to receiving the request.
 50. The system of claim48, wherein the operations further comprise: receiving negative responsedata from the content item exchange, the negative response dataspecifying that the content item exchange will not provide an exchangecontent item for a price that meets the reserve price; and in responseto receiving negative response data, selecting a content item that isprovided by one of the reservation content item providers as the contentitem to be provided in response to receiving request.
 51. The system ofclaim 50, wherein the operations further comprise: in response toreceiving negative response data, selecting one of the reservationcontent item providers as a winning content item provider, the winningcontent item provider being the reservation content item providercorresponding to a highest impression score relative to all otherreservation content item providers; and wherein selecting a content itemthat is provided by one of the reservation content item providerscomprises selecting a content item provided by the winning content itemprovider.
 52. The system of claim 50, wherein the operations furthercomprise: in response to receiving negative response data: determiningthat two or more of the reservation content item providers eachcorrespond to a same impression score that is a highest impression scorerelative to all other reservation content item providers; assigning eachof the two or more reservation content item providers an assignmentprobability based on one or more characteristics associated with therequest; and selecting one of the two or more reservation content itemproviders based on the assignment probabilities; and wherein selecting acontent item that is provided by one of the reservation content itemproviders comprises selecting a content item that is provided by theselected reservation content item provider.
 53. The system of claim 50,wherein the operations further comprise: in response to receivingnegative response data: determining that two or more of the reservationcontent item providers each correspond to a same impression score thatis a highest impression score relative to all other reservation contentitem providers; assigning each of the two or more reservation contentitem providers an assignment probability based on a particularcombination of the two or more reservation content item providers thateach correspond to the same impression score; and selecting one of thetwo or more reservation content item providers based on the assignmentprobabilities; and wherein selecting a content item that is provided byone of the reservation content item providers comprises selecting acontent item that is provided by the selected reservation content itemprovider.
 54. The system of claim 48, wherein the operations furthercomprise: determining that a number of remaining resource requests forthe reservation period exceeds a number of unfulfilled impressionreservations for the reservation period; and wherein the providing areserve price is only performed in response to determining that thenumber of remaining resource requests exceeds the number of unfulfilledimpression reservations.
 55. The system of claim 48, wherein the reserveprice is determined based on an impression score for at least one of thereservation content item providers, each impression score specifying,for a reservation content item provider, a measure of quality of theimpression to the reservation content item provider.
 56. Acomputer-readable medium encoded with instructions that when executed bya data processing apparatus cause the data processing apparatus toperform operations comprising: receiving, during a reservation period, arequest for a resource that includes a content item slot that has beenreserved by one or more reservation content item providers; determiningan expected revenue that will be obtained by selecting one of thereservation content item providers to provide a content item in responseto receiving the content item request data; and providing to a contentitem exchange, a reserve price that is based on the expected revenue.